Text Input Window with Auto-Growth

ABSTRACT

A user input panel dynamically expands to accommodate user input, such as handwritten or keyboard input. Expansion may occur in one or two out of four possible directions, depending upon the language to be written or typed. For example, when writing English words, the input panel may expand to the right as the user writes and then downward when the input panel has fully expanded rightward.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 10/704,127, filed Nov. 10, 2003, which is herebyincorporated by reference as to its entirety.

FIELD OF THE INVENTION

Aspects of the present invention are directed generally to improved userinput interfaces in computer systems, and more particularly to providingan input panel that dynamically expands on an on-demand basis as theuser writes or types.

BACKGROUND

User input devices and interfaces have been growing rapidly to meet thedemand of new types of computing devices. More recently, tablet-basedpersonal computers and hand-held computers have become popular. Thesedevices typically have a writing surface that converts movement of astylus across the surface into electronic ink. This ink can berecognized and converted into text, or can be stored in an electronicink format.

For example, the Microsoft WINDOWS brand XP Tablet PC Edition operatingsystem provides a data entry user interface, sometimes referred to asthe text input panel, the Tablet PC input panel or “TIP,” through whicha user can employ a stylus to enter data into a computer. This type ofuser interface falls into the broader category of data entry graphicaluser interfaces, and may be referred to as a data input panel. Some datainput panels also may provide a “soft” keyboard surface that displaysthe characters of a conventional keyboard. If the user employs a stylusto tap on the display of a key with this type of keyboard, then thecomputer will receive the character associated with that key as inputdata. Other types of data input panels may have specialized inputsurfaces. For example, some user interfaces may provide a plurality ofletter-sized writing surfaces for receiving characters in an East Asianalphabet as input.

While these types of data input panel increase the usefulness ofcomputers that employ a stylus input device, there are still someinconveniences associated with conventional data input panels. Forinstance, data input panels are generally specifically invoked by a userin a manner that is difficult or inconvenient. In order to enter datainto an application using a data input panel, the user manuallyactivates the data input panel from outside of the application. Manycomputer users are unfamiliar with data input panels, however, and mayoverlook or not know how to invoke this resource, rendering such aresource effectively “hidden” from the novice user.

Additionally, because the data input panel is a graphical user interfaceinvoked separately from the application, it typically is displayedseparately from the application as well. In some situations, the userinterface may be overlaid onto the displayed application itself. Whilethis arrangement allows the user to enter data close to its destinationin the application, the user interface may inadvertently obscure thedestination or adjacent destinations. Even if the user interface doesnot initially obscure a desired destination for entered data, however,the user may need to constantly relocate the user interface as new datais entered into the application, to prevent the user interface fromobscuring the new insertion points for that data. With still other typesof data input panels, the user interface may be “docked” in a spacebelow the application. While positioning the data input panel in thisfashion ensures that it will not prevent the user from viewing a desireddestination for new data, it is inconvenient for the user tocontinuously move the stylus back-and-forth between the application andthe data input panel to both control the application and enter data intothe application.

SUMMARY OF THE INVENTION

Aspects of the present invention allow a user to easily and efficientlyenter user input in a way that is less intrusive. An input panel isprovided that may be easily discoverable by the novice user and mayrequire little manipulation on the part of the user to be effectivelyused. The input panel may provide easier access to handwritten input,keyboard input, and other user input when and where needed by the user,while possibly reducing the likelihood of obscuring certain importantdisplayed elements that may be of interest to the user.

Further aspects of the invention are directed to causing an input panelto appear in a convenient location and to dynamically expand toaccommodate the user input. The input panel may expand in one or twodirections, such as rightward and downward, without expanding in theother two directions (upward and leftward). This may be particularlyuseful where a language such as English is written or typed into theinput panel, for example where the language is likewise writtenrightward in a horizontal row and then downward in succeeding rows. Thedirection(s) that the input panel expands may depend upon the chosenlanguage.

According to a further aspect of the invention, the input panel mayexpand in a way that respects certain defined margins. For example, theinput panel may expand only up to a certain margin distance away from anedge of a display. This may protect the availability of often usefuluser interface elements near the edges of the display.

Still further aspects of the invention are directed to convenient andintuitive summoning and dismissal of the input panel, while reducing theprobability that an uninvited input panel will appear or that a desiredinput panel will inadvertently disappear.

These and other aspects of the invention will become apparent to one ofordinary skill in the art upon a reading of the following description,drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The foregoing summary of the invention, as well as the followingdetailed description of illustrative embodiments, is better understoodwhen read in conjunction with the accompanying drawings, which areincluded by way of example, and not by way of limitation with regard tothe claimed invention.

FIG. 1 is a functional block diagram of an illustrative computingenvironment that may be used in accordance with at least one aspect ofthe present invention.

FIG. 2 is a plan view of an illustrative computing environment that maybe used in accordance with at least one aspect of the present invention.

FIG. 3 is a flow chart showing illustrative steps that may be performedin accordance with at least one aspect of the present invention.

FIGS. 4, 5, 7, and 8 are various screen shots showing how an input panelmay expand in accordance with at least one aspect of the presentinvention.

FIGS. 6 and 13 are depictions of illustrative input panels in accordancewith at least one aspect of the present invention.

FIGS. 9 through 11 are various screen shots showing how an input panelmay be used in connection with an input area in a software application,in accordance with at least one aspect of the present invention.

FIG. 12 is a screen shot showing how an input panel may be used inconnection with an Internet web browser software application, inaccordance with at least one aspect of the present invention.

DETAILED DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

General Computing Environment

FIG. 1 is a functional block diagram of an example of a conventionalgeneral-purpose digital computing environment that can be used toimplement various aspects of the present invention. In FIG. 1, acomputer 100 includes a processing unit 110, a system memory 120, and asystem bus 130 that couples various system components including thesystem memory to the processing unit 110. The system bus 130 may be anyof several types of bus structures including a memory bus or memorycontroller, a peripheral bus, and a local bus using any of a variety ofbus architectures. The system memory 120 includes read only memory (ROM)140 and random access memory (RAM) 150.

A basic input/output system 160 (BIOS), containing the basic routinesthat help to transfer information between elements within the computer100, such as during start-up, is stored in the ROM 140. The computer 100also includes a hard disk drive 170 for reading from and writing to ahard disk (not shown), a magnetic disk drive 180 for reading from orwriting to a removable magnetic disk 190, and an optical disk drive 191for reading from or writing to a removable optical disk 192 such as a CDROM or other optical media. The hard disk drive 170, magnetic disk drive180, and optical disk drive 191 are connected to the system bus 130 by ahard disk drive interface 192, a magnetic disk drive interface 193, andan optical disk drive interface 194, respectively. The drives and theirassociated computer-readable media provide nonvolatile storage ofcomputer readable instructions, data structures, program modules andother data for the personal computer 100. It will be appreciated bythose skilled in the art that other types of computer readable mediathat can store data that is accessible by a computer, such as magneticcassettes, flash memory cards, digital video disks, Bernoullicartridges, random access memories (RAMs), read only memories (ROMs),and the like, may also be used in the example operating environment.

A number of program modules can be stored on the hard disk drive 170,magnetic disk 190, optical disk 192, ROM 140 or RAM 150, including anoperating system 195, one or more application programs 196, otherprogram modules 197, and program data 198. A user can enter commands andinformation into the computer 100 through input devices such as akeyboard 101 and pointing device 102. Other input devices (not shown)may include a microphone, joystick, game pad, satellite dish, scanner orthe like. These and other input devices are often connected to theprocessing unit 110 through a serial port interface 106 that is coupledto the system bus, but may be connected by other interfaces, such as aparallel port, game port or a universal serial bus (USB). Further still,these devices may be coupled directly to the system bus 130 via anappropriate interface (not shown). A monitor 107 or other type ofdisplay device is also connected to the system bus 130 via an interface,such as a video adapter 108. In addition to the monitor, personalcomputers typically include other peripheral output devices (not shown),such as speakers and printers. In a preferred embodiment, a pendigitizer 165 and accompanying pen or stylus 166 are provided in orderto digitally capture freehand input. Although a direct connectionbetween the pen digitizer 165 and the serial port is shown, in practice,the pen digitizer 165 may be coupled to the processing unit 110directly, via a parallel port or other interface and the system bus 130as known in the art. Furthermore, although the digitizer 165 is shownapart from the monitor 107, it is preferred that the usable input areaof the digitizer 165 be co-extensive with the display area of themonitor 107. Further still, the digitizer 165 may be integrated in themonitor 107, or may exist as a separate device overlaying or otherwiseappended to the monitor 107.

The computer 100 can operate in a networked environment using logicalconnections to one or more remote computers, such as a remote computer109. The remote computer 109 can be a server, a router, a network PC, apeer device or other common network node, and typically includes many orall of the elements described above relative to the computer 100,although only a memory storage device 111 has been illustrated inFIG. 1. The logical connections depicted in FIG. 1 include a local areanetwork (LAN) 112 and a wide area network (WAN) 113. Such networkingenvironments are commonplace in offices, enterprise-wide computernetworks, intranets and the Internet.

When used in a LAN networking environment, the computer 100 is connectedto the local network 112 through a network interface or adapter 114.When used in a WAN networking environment, the personal computer 100typically includes a modem 115 or other means for establishing acommunications over the wide area network 113, such as the Internet. Themodem 115, which may be internal or external, is connected to the systembus 130 via the serial port interface 106. In a networked environment,program modules depicted relative to the personal computer 100, orportions thereof, may be stored in the remote memory storage device.

It will be appreciated that the network connections shown areillustrative and other techniques for establishing a communications linkbetween the computers can be used. The existence of any of variouswell-known protocols such as TCP/IP, Ethernet, FTP, HTTP and the like ispresumed, and the system can be operated in a client-serverconfiguration to permit a user to retrieve web pages from a remoteweb-based server. Any of various conventional web browsers can be usedto display and manipulate data on web pages.

FIG. 2 shows an example of a stylus-based computer processing system(also referred to as a tablet PC) 201 that can be used in accordancewith various aspects of the present invention. Any or all of thefeatures, subsystems, and functions in the system of FIG. 1 can beincluded in the computer of FIG. 2. Tablet PC 201 includes a largedisplay surface 202, e.g., a digitizing flat panel display, preferably,a liquid crystal display (LCD) or OLED screen, plasma display and thelike, on which a plurality of windows 203 is displayed. Using the tip ofa stylus 204 (the tip also being referred to herein as a “cursor”), auser can select, highlight, and write on the digitizing display area.Examples of suitable digitizing display panels include electromagneticpen digitizers, such as the Mutoh or Wacom pen digitizers. Other typesof pen digitizers, e.g., optical digitizers, may also be used. Tablet PC201 interprets marks made using stylus 204 in order to manipulate data,enter text, and execute conventional computer application tasks such asspreadsheets, word processing programs, and the like.

The stylus 204 may be equipped with buttons or other features to augmentits selection capabilities. In one embodiment, a stylus could beimplemented as a “pencil” or “pen”, in which one end constitutes awriting portion and the other end constitutes an “eraser” end, andwhich, when moved across the display, indicates portions of the displayare to be erased. Other types of input devices, such as a mouse,trackball, or the like could be used. Additionally, a user's own fingercould be used for selecting or indicating portions of the displayedimage on a touch-sensitive and/or proximity-sensitive display.Consequently, the term “user input device”, as used herein, is intendedto have a broad definition and encompasses many variations on well-knowninput devices.

Input Panel Expansion Behavior

Referring to FIG. 3, in step 301, a computer (e.g., computer 100 or 201)may check the position of a user input device (e.g., stylus 166 or 204)relative to an input surface (e.g., digitizer 165 or display surface202). The user input device will be illustratively considered in thefollowing embodiments to be the stylus 204. However, any user inputdevice may be used, such as the mouse 102. In step 302, the computer maydetermine, based on the position of the stylus 204, whether the stylus204 is hovering over the input surface and whether the stylus 204 ishovering over a valid “input area.” The term “hovering” means that thestylus 204 (particularly, for example, the tip of the stylus 204) isproximate to, but not physically touching, the input surface. Athreshold-based determination may be made. For example, the computer maydetermine whether the tip of the stylus is within a predeterminedorthogonal distance from the input surface. If so, then the stylus ishovering. If not (e.g., the stylus 204 is too far away from the inputsurface or is touching the input surface), then the stylus 204 is nothovering.

The term “input area” means an area of the display or user input surfacethat accepts valid user input. At any given time, there may be one ormore input areas, and these input areas may change, move, appear, anddisappear over time. Which areas are input areas may depend upon whichapplications are running, which operating system is running, whichapplication(s) has focus (i.e., is the application currently andactively being interacted with by the user), and/or other factors. Inputareas typically accept typed text from a keyboard and/or handwritten inkfrom a stylus. In some embodiments, the only input area may be whereverthe “system caret” is currently located, or at least a predefined zonearound and near the system caret. A system caret is the insertion pointthat the user is currently working with. The insertion point may beindicated to the user via a cursor such as a blinking text cursor.Another “cursor” may also exist that indicates the location on thedisplay that the stylus 204 (or other input device such as the mouse102) is interacting with (e.g., the location on the display over whichthe stylus 204 is hovering, or the location on the display that thestylus 204 is touching). The cursor indicating the insertion point isnot necessarily the same as the cursor indicating the stylus 204interaction point.

In step 303, if the stylus 204 is not hovering, then any invocationtarget that may be displayed (as discussed below) is removed from beingdisplayed, the computer then checks again in step 301 whether the stylus204 is hovering. This loop is repeated until it is determined that thestylus 204 is hovering. Once this occurs, then in step 304 it isdetermined whether the stylus 204 has either hovered for a predeterminedtimeout period or whether any displayed invocation target has beenselected. The invocation target may be selected by, e.g., clicking onthe invocation target such as with a button of the mouse 102, by tappingon the invocation target with the stylus 204, by hovering over theinvocation target, or by pressing a button on the computer or the stylus204. If neither of these determinations occurs, then in step 305, aninvocation target is displayed (or is continued to be displayed ifalready displayed), and the computer continues to check whether thestylus is hovering in steps 301 and 302. The invocation target may bedisplayed immediately after the stylus has begun hovering over anappropriate area of the input surface or after a time delay. Step 304may include both a timeout determination and an invocation targetselection determination, or it may only include one of these twodeterminations. Although the present illustrative embodiment isdescribed as displaying the invocation target in response to the stylus204 hovering, the invocation target may also be displayed in response toother user input, such as in response to the stylus 204 actuallycontacting the input surface or in response to some other gestureperformed by the stylus 204. It will be understood that hovering asdiscussed herein is presented only as an illustrative user input.

An “invocation target” is an indication to the user that an input panelis available and can be invoked. The invocation target may be anindication that is visible and/or audible to the user, such as an icondisplayed on a display, an audible sound such as a beep or click, or amodified cursor. An example of an invocation target 403 is shown in FIG.4. The invocation target 403 may be displayed at a predeterminedlocation on a display 401 (which may be part of the input surface), orat a location that depends upon the location 402 on the display 401 overwhich the stylus 204 is hovering or otherwise interacting with thedisplay 401 (the “interaction point” 402). For example, the invocationtarget 403 may be displayed at a location that is displaced from theinteraction point 402 by a predetermined amount. Alternatively, theinvocation target 403 may be displayed at a location that depends uponthe location of the insertion point in the active input area. Forexample, the invocation target 403 may be displayed at a location thatis displaced from the insertion point by a predetermined amount. In theshown embodiment, the invocation target 403 is displayed at a locationwhere the edge of the invocation target 403 nearest the interactionpoint 402 is displaced on the display 401 by X distance in thehorizontal direction and by Y distance in the vertical direction, whereX and Y may be the same or different and may be negative or positive. Itshould be noted that in this example, the “X”, “Y”, and broken lines asshown are not actually displayed but are included only for explanatorypurposes. The X and Y distances may be any amount, such as about ¼ of aninch or less, about ⅓ of an inch, or about ½ of an inch or more.

Although the invocation target 403 as shown is a box with an “X” inside,this is merely an example. The invocation target 403 may appear to theuser in any form, such as an icon, an animation, an image, text, and/orany other type of visual element(s). Also, although the invocationtarget 403 is shown as a rectangle, it may be of any shape such as acircle, square, triangle, and/or any other geometric or non-geometricshape.

As discussed above, the user's stylus 204 may hover for the timeoutperiod, or the user may select the invocation target 403. The user mayselect the invocation target 403 in any of a number of ways, such as bytapping on the displayed invocation target 403 with the stylus 204. Inresponse to either of the actions set forth in step 304 occurring, aninput panel is presented to the user in step 306. The input panel isable to receive user input, such as electronic ink input from the stylus204, typewritten input from a keyboard, and/or other input.

An example of a displayed input panel 501 is shown in FIG. 5. The inputpanel 501 may replace the invocation target 403 or it may be displayedin addition to the invocation target 403. The input panel may include ananchor point 502, which may or may not be displayed. The input panel 501may be displayed at any location on the display 401, such as where itsanchor point 502 is at a predetermined location or at a location thatdepends upon the interaction point 402 and/or the location of theinvocation target 403. In the example shown, the input panel 501 isdisplayed in the same, or nearly the same, location as the invocationtarget 403 that was previously displayed before being replaced by theinput panel 501.

Referring back to FIG. 3, in step 307 the input panel 501 may receive asuser input any type of content desired, such as text, ink, images, audio(e.g., speech), etc. However, the illustrative input panel 501 in theembodiments disclosed will be considered to be primarily for receivingtext, ink, and/or speech content. The reason for this is the input panel501 may, in certain embodiments, be considered a text input panel forreceiving meaningful input for entry into a currently runningapplication such as a word processing application. As can be seen inFIG. 5, the letters “abc” are written in handwritten ink as content inthe input panel 501. This may be done by, e.g., using the stylus 204 towrite “abc” on the input surface, and more particularly on the portionof the input surface corresponding to the input panel 501.

In FIG. 3, in step 308, the computer may check whether the user input isnear or touching a boundary of the input panel 501. Boundaries of theinput panel 501 may be coextensive with the visible edges of the inputpanel 501, may be within the visible representation of the input panel501, and/or may extend beyond the visible representation of the inputpanel 501. For example, a right boundary may be the same as a rightvisible edge of the input panel 501, or the right boundary may be apredetermined distance to the left of the right visible edge. FIG. 6shows an example of a right boundary 606 that is to the left of theright edge 602 of the input panel 501. However, the boundary 606 mayalso be at the same location as the right edge 602. Also indicated arethe top edge 601, left edge 605, and bottom edge 603 of the input panel501.

FIG. 7 shows what may occur in response to the user input being near ortouches a boundary of the input panel 501. FIG. 7 also shows directions“up,” “down,” “left,” and “right.” These illustrative directions will beused consistently throughout the specification to help explain aspectsof the invention, and are measured from the user's point of view as theuser views the display 401. The directional arrows and direction text asshown are not necessarily displayed on the display 401, but are merelyshown for explanatory purposes. The directions as used herein are alsorelative to the display 401 and are not necessarily relevant to ordescriptive of the pull of Earth's gravity. For example, the display 401may lay horizontally on a table, yet the “up” direction as shown remainsthe “up” direction relative to the display 401.

As can be seen, the user input, originally “abc”, has been changed tonow read “abc def ghi jkl mno pqr”. Because this user input would nothave fit in the input panel 501 as shown in FIG. 5, the input panel 501automatically expands to accommodate the user input. In this embodiment,responsive to the user input being near or touching a right boundary ofthe input panel 501, the input panel 501 dynamically expands its rightvisible side 602 and/or right boundary 606 as necessary to fit the userinput. For instance, if the user input had only been “abc def ghi”, thenthe input panel 501 may have expanded rightward less than shown in FIG.7. In real time, as the user input moves toward the right, the inputpanel 501 may continue to expand rightward. However, the input panel 501may be limited as to how much it can expand rightward. A physicallimitation would be the boundary of the display 401. The input panel 501may further be artificially limited in its rightward expansion. Forexample, the input panel 501 may only be able to expand rightward up toa predetermined horizontal right margin Mhr from the right boundary ofthe display 401. The margin Mhr may be fixed or variable, and may be setby the user and/or by an application. Margin Mhr may be measured asabsolute distance or as a percentage of the size of the display 401. Forexample, Mhr may be about ¼ inch, about ½ inch, between about ¼ inch andabout ½ inch, about 3% of the display 401 boundary width, about 5% ofthe display 401 boundary width, or between about 3% and about 5% of thedisplay 401 boundary width. However, Mhr may be other values. Throughoutthe expansion process, the anchor point 502 may remain fixed. This meansthat the top edge 601 and the left edge 605 of the input panel 501 mayalso remain fixed throughout the expansion process.

Thus, referring back to FIG. 3, if in step 309 the input panel 501 hasnot yet fully expanded rightward to its limit, then in step 310 theinput panel 501 expands rightward and continues to accept further userinput in step 307. The input panel 501 may expand rightward inincrements. The increments may be of a fixed size or may vary dependingupon the user input. Where the increment size is fixed, the incrementmay be about 1.5 inches, or about 1.75 inches or about 2 inches, or moreor less than these amounts.

On the other hand, if in step 309 it is determined that the input panel501 has already reached its rightward limit, then the input panel 501may expand in another direction in step 312. In the shown embodiment,the other direction is down. The input panel 501 may then continue toaccept further user input in step 307. FIG. 8 illustrates what mayhappen when the input panel 501 has reached its rightward limit and theuser continues to provide user input to the input panel 501. In thiscase, the user input has now increased to become “abc def ghi jkl mnopqr stu vwx yz1 234”. This entire user input could not fit in the inputpanel 501 as shown in FIG. 7, so the input panel 501 has automaticallyexpanded downward to accommodate the additional user input. In otherwords, the bottom edge 603 of the input panel 501 has moved downward inresponse to rightward extension of an ink stroke. This rightward anddownward expansion of the input panel 501 may occur in real time whilethe ink is being written. Note that the right margin Mhr is stillrespected. The downward expansion may begin only after the rightwardexpansion has cause the input panel 501 to expand fully to the right upto margin Mhr. Alternatively, at least some downward expansion may occurprior the input panel 501 expanding fully to the right, before itexpands to margin Mhr. Moreover, at least some downward expansion mayoccur simultaneously with rightward expansion (which may appear to theuser as effectively a diagonal expansion).

However, in the present embodiment, the input panel 501 may expanddownward only if it has not yet fully expanded in the downwarddirection. Thus, in step 311, if the input panel 501 has not yet fullyexpanded downward, then it may expand in an incremental manner. Forexample, the input panel 501 may expand by a predetermined increment orby an amount depending upon the user input. In one embodiment, thedownward incremental expansion amount may be a fixed amount of about 1.5inches, or about 1.75 inches or about 2 inches, or more or less thanthese amounts. In another embodiment, the downward incremental expansionamount may depend upon the vertical size of the handwritten or verticalcharacters that has been previously written into the user input panel.The larger the previously written text, the larger the downwardincrement, with the expectation that further user input characters willbe of a similar size.

On the other hand, if the input panel 501 has already expanded to amaximum downward limit as determined in step 311, then the input panel501 may not expand further. However, the input panel 501 may continue toaccept user input. The expansion limit in the downward direction may belimited by the bottom boundary of the display 401. The downwardexpansion limit may further be limited by an artificial vertical bottommargin Mvb. Margin Mvb may be determined in the same manner as marginMhr.

At this point, it should be noted that the anchor point 502 in thisembodiment continues to remain in the same fixed location on the display401 throughout FIGS.—5, 7, and 8. This is true even though the inputpanel 501 may have fully expanded in the rightward and/or downwarddirections. In some embodiments, the user may be able to manually movethe anchor point (such as by dragging/dropping it) to make further roomfor input panel expansion. The anchor point 502 may represent theintersection of the upper and left boundaries of the input panel 501.However, the anchor point 502 may represent the intersection of otherboundary combinations of the input panel 501, such as the intersectionof the upper and right boundaries of the input panel 501. Returning tothe example where the anchor point 502 represents the intersection ofthe upper and left boundaries, the upper and left boundaries of theinput panel 501 may thus remain fixed along with the anchor point 502.Therefore, once the input panel 501 has fully expanded in both therightward and downward directions, the input panel 501 cannot expandfurther. As will be discussed further, which intersection of boundariesthe anchor point 502 represents may depend upon which language isexpected to be written or typed into the input panel 501.

Although the input panel 501 may reach a point where it can no longerexpand in any direction (i.e., where the outcome of step 311 is “yes”),the user may continue to add user input and/or edit existing user inputin the existing area defined by the input panel 501. Where the inputpanel 501 can no longer expand, the input panel 501 may in responsedevelop one or more scroll bars in the horizontal and/or verticaldirections for allowing still further user input to be entered. However,in some embodiments, it may be desirable to keep the interface with theinput panel 501 as simple as possible, and so scroll bars would notbecome available in that instance.

A horizontal left margin Mhl and a vertical top margin Mvt are alsoshown in FIG. 8. These margins are artificial margins that may be usedsuch that the input panel 501 cannot cross those margins. In someembodiments, the anchor point 502 can never be located to the left ofmargin Mhl, nor can the anchor point 502 ever be located above marginMvt. This would be true regardless of where the interaction point 402is. Any rules defining where the input panel 502 should be displayedshould ensure that the input panel 502 is not located outside of marginsMhl, Mhr, Mvt, and Mvb, both initially and while the input panel 502expands. If necessary, the initial default location of the input panel502 that would otherwise be used as described previously may be modifiedby an amount sufficient to ensure that the input panel 502 does notviolate these margins. Whenever the default placement rules do notresult in placing the invocation target and/or the input panel withinthe margins, then the invocation target and/or input panel may be movedas little as possible so as to be as close as possible to the defaultlocation without violating the margins.

Input Panel Interaction With Input Areas

Examples of how the input panel 501 may be used is shown in FIGS. 9through 11. In FIG. 9, the display 401 shows the graphical userinterface representation 901 of an open software application, such as,but not limited to, a word processing application, spreadsheetapplication, Internet web browser, or an email program. The softwareapplication may display content 902, which is text in this example butmay be other content. The content 902 may be interacted with, such as byediting, deleting, or adding to it. Therefore, in this particularembodiment, an area 903 on the display 401 (shown by broken lines thatare not necessarily displayed) containing the content 902 may beconsidered an input area as previously defined. The stylus 204 may behovering over the display 401 in an area indicated by the interactionpoint 402 (or over a portion of an input surface corresponding with theinteraction point 402 on the display 401, where the input surface is notthe same as the display 401). Because the interaction point 402 iswithin the input area 903, an invocation target, and eventually theinput panel 501, may be displayed as previously described. Had theinteraction point 402 been outside of the input area 903 (and outside ofany input area), then an invocation target, and the input panel 501, maynot have been displayed. Thus, whether the invocation target and/orinput panel are displayed may depend upon where the interaction point402 is relative to an input area. Input areas may be defined by theoperating system and/or by the application software currently running.

In this example, the input panel 501 may be associated with the inputarea 903 and may initially be empty of user input and content. The inputpanel 501 may particularly be associated with an insertion point in theinput area 903. Typically, the application, or a system hosting theapplication (such as the operating system), will provide an insertionpoint where data created by an input device will be inserted into acontrol such as the input area 903. For example, an application maydisplay an insertion point to indicate where data created by an inputdevice will be inserted. An insertion point for text typically may bedisplayed as a cursor in the form of a blinking vertical line. Theposition of the insertion point will conventionally be controlled by oneor more input devices. The insertion location may further be a selectionof replaceable text. When a user wishes to replace a selection of textwith new input data, the user may select the entire text to be replaced.For example, a user may activate a command button associated with apointing device, and then move the pointing device so that a displayedpointer passes over or through the text to be replaced. This process issometimes referred to as “blocking” a selection of text. When a userthen enters data into the control, the entirety of the selected text isreplaced with the newly entered data.

In this example, the user has blocked the word “three” in the content902, with the intention of editing that word. The selected portion maytherefore be considered the insertion point 904 in the input area 903.Any changes that the user makes to the content 902 using the input panel501 would occur at the insertion point 904. As shown, the user addshandwritten user input “at” to the input panel 501. The insertion point904 associated with the input panel 501 may be further highlighted orotherwise modified to indicate that it is associated with an inputpanel.

Referring to FIG. 10, the user continues to write, adding to the “at”previously written, so that the handwritten content in the input panel501 is now “at least four, but no more than five”. As previouslydescribed, the input panel 501 may expand rightward, and if necessarydownward, to accommodate the user input. Again, the input panel 501 mayexpand rightward and downward in such a way that it does not violate theMhr and Mvb margins and that maintains the anchor point 502 in a fixedposition. In this example, the anchor point 502 is the upper left cornerof the input panel 501. Also, the input panel 501 may initially appearin such a location that does not violate the Mhl, Mhr, Mvt, and Mvbmargins.

As shown in FIG. 11, when the user is finished adding user input to theinput panel 501, the user input may be transferred to the input areaassociated with the input panel 501, which in this example is input area903. In particular, the user input may be transferred to a locationwithin the input area 903 associated with the interaction point 402,which in this case is the insertion point 904 generated by the activesoftware application. The user input may further be converted intoanother form prior to adding it to the input area 903. For example,where the user input is handwritten ink, the ink may be recognized intotext, and the recognized text may be added to the input area 903. Asshown, the user's handwriting is recognized, and the resulting textcorresponding to the handwriting replaces the highlighted word “three”at the insertion point 904, such that the relevant portion of thecontent 902 now reads, “ . . . within at least four, but no more thanfive, Years . . . ”. After insertion of the recognized user input fromthe input panel 501, the insertion point 902 is now located after theinserted content, and is shown in the form of a standard text cursor.

Handwriting recognition may be performed automatically or manually atvarious times as desired. For example, handwriting recognition may beinvoked only after the user is done with adding the user input, orautomatically while the user is adding the user input. Where recognitionis performed while the user is adding input, handwriting recognition maybe automatically performed after each character is written or after eachword is written. The text resulting from handwriting recognition may beadded to the input area 903 while the user is still adding input withoutwaiting for the user to finish, or after the user has finished.Recognition may occur in response to the stylus 204 being lifted off theinput surface for at least a certain period. Recognition mayalternatively occur in response to a manual command from the user, inresponse to switching input modes, or and/or in response to switchingapplications.

There may be exceptions to the rule that the anchor point 502 is fixed.For example, where the insertion point 904 is moved under the inputpanel 501, the input panel 501 may move a sufficient amount to preventthe insertion point 904 from being hidden from the user. The insertionpoint may move under the input panel 501 where, for example, the userinput handwriting in the input panel 501 is recognized and sent to theinput area 903 while the user writes.

FIG. 12 illustrates another example of using an input panel inconnection with an Internet web browser software application. Thebrowser application may display a browser window 1202, with a contentbox 1203 in the window for displaying the content of a currently-browsedweb page. The browser application may further display an Internetaddress input box 1201, which may receive user-input Internet addressessuch as in the form of universal resource locators (URLs). In thisembodiment, the Internet address input box 1201 is an input area forpurposes of interacting with an input panel. Another input area 1205 isshown in the content box 1203. The existence and/or location(s) of anyinput area(s) in the content box 1203 may be defined by and depend uponthe particular Internet web page being browsed at any given time. Thedisplayed Internet web page may be considered a document, and ingeneral, any type of document may define the existence and/orlocation(s) of any input area(s) therein.

If the interaction point (not shown) remains in or near the Internetaddress input box 1201, then in response an invocation target (notshown) and/or an input panel 1204 may appear. In this example, the hoverposition may be in the Internet address input box 1201, which is outsidethe Mvt margin. Therefore, placement rules should ensure that theinvocation target and/or the input panel 1204 do not display above theMvt margin. Depending upon the specific circumstance, the invocationtarget and/or the initially-displayed input panel 1204 may need to bemoved downward by a small amount to ensure this. In addition, thelocation that the invocation target and/or the input panel 1204 aredisplayed may be affected by any auto-complete dropdowns that areexpected to show. For example, where the Internet address input box 1201has an auto-complete feature that drops down a list of suggestedcomplete Internet addresses, the invocation target and/or input panel1204 initial display position may be adjusted to be displayed belowwhere the bottom of the expected dropdown auto-complete list would bedisplayed.

Also, depending upon how far to the right of the display the interactionpoint is located, the invocation target and/or the initially-displayedinput panel 1204 may need to be moved leftward to ensure that thedefault initial size of the input panel 1204 does not violate the Mhrmargin. The user may write, using the stylus 204, a URL such“www.microsoft.com”. This handwritten URL may be recognized, and theresulting text may be added to the Internet address input box 1201. Inresponse, the web page associated with that URL may be browsed to. Hadthe interaction point been associated with the input area 1205 (e.g.,inside the input area 1205), the resulting user input sent to the inputarea 1205 may interact with the displayed web page and/or be sent to aremote server providing the displayed web page.

Summoning and Dismissing

Input panels have been described as being summoned by hovering thestylus 204 over an input area. However, an input panel (such as theinput panel 501 or the input panel 1204) may be summoned in any of avariety of ways. For example, an input panel may be explicitly summonedin response to a user's commands, such as a keyboard command, menucommand, or gesture of the stylus 204. Stylus gestures may include anytype of gesture such as hovering, tapping, and/or dragging of the stylus204 relative to the display 401. An input panel may be summoned with orwithout a preceding invocation target.

Moreover, input panels have been described as being summoned after aninvocation target first appears. However, the invocation target is notnecessary; an input panel may be summoned without a prior invocationtarget. This may be particularly useful where the input area iscurrently blank (devoid of user input) or is of a certain type. Thus, insome embodiments, an invocation target may normally first appearfollowed, if summoned, by the input panel, while in certain situationsin the same embodiments the input panel may be summoned without theinvocation target. For example, the input panel may be immediatelysummoned in response to the stylus 204 hovering over a blank input area.Or, the input panel may be immediately summoned in response to thestylus 204 hovering over or approaching a blank “To,” “From,” or“Subject” input area in an email. The input panel that is summonedimmediately may differ from the input panel that is summoned after aninvocation target is displayed. For example, the input panel that issummoned immediately may be smaller or missing certain elements (such askeyboard buttons, mode switching user interfaces, etc.). This smaller,simpler state may be preferable in such a situation because thissmaller, simpler input panel would be an educated guess that the useractually desires to start inputting via the input panel, whereas showingthe full input panel in such a situation may be a bit intrusive.

Likewise, an input panel (such as the input panel 501 or the input panel1204) may be dismissed in any of a variety of ways. For example, aninput panel may be explicitly dismissed by a user-invoked keyboardcommand, menu command, or gesture of the stylus 204. Again, such agesture may be any type of gesture. Once closed explicitly, theinvocation target and/or the input panel may not be automaticallydisplayed again in connection with the same input area or softwareapplication. However, if the focus changes or if the stylus 204 isremoved from the input surface in the currently-focused input area orapplication, then automatic invocation of the invocation target and/orinput panel may again be performed in connection with that same inputarea.

In one example of a gesture-based dismissal, if the stylus 204 hoversoutside the displayed input panel (or at least a predetermined distance,such as ½ inch, outside a boundary of the input panel), then in responsethe input panel may be dismissed. A dialogue window may provide the userthe opportunity to verify that dismissal is desired. This “hover away”dismissal may always dismiss the input panel, or may only dismiss theinput panel if the stylus 204 has already entered the area of the inputpanel. This may reduce unintended dismissal from occurring. Anotherexample of gesture-based dismissal may be that the input panel isdismissed in response to the stylus 204 going out of range of the inputsurface. This “out of range” dismissal may always work or may work onlyif user input has already been provided to the input panel. Again, thelatter may help reduce unintended dismissal. The “hover away” dismissaland the “out of range” dismissal may also each require that the stylus204 be away from the input panel or out of range for at least a minimumtimeout. The timeout may be any amount of time, such as about one secondor more.

Yet another way to dismiss an input panel is to dismiss the softwareapplication and/or input area associated with the input panel. Shouldthis occur, any user input in the input panel not already sent to thesoftware application and/or input area may be flushed and not sent.However, simply changing the focus to another software application or tothe shell may not affect the input panel, which may remain open but outof focus until selected again.

Input Panel Content Formats

Examples of input panels have been described thus far in connection withhandwritten user input using a stylus. However, an input panel mayaccept a variety of user input formats including keyboard input andspeech. Depending upon the type of user input desired, the input panelmay be displayed in a different form. For example, where freestylehandwriting is desired, the input panel may be blank or may contain asingle line on which to write words. Where character-by-characterrecognition is desired, the input panel may display a series of blocksor the like, one for each character, such as is shown in FIG. 13. As theinput panel expands, additional character blocks may be displayed. Wherekeyboard input is desired, the input panel may display an icon of akeyboard, or even a graphical interactive keyboard that may be used toenter characters. The input panel may include one or more buttons orother controls 1301 allow the user to manually select which type ofinput panel to display. For example, the buttons 1301 may allow the userto select between keyboard, freestyle handwriting, andcharacter-by-character (block style) content input panels.Alternatively, the user may select the type of input panel content fromthe invocation target.

Multiple Language Accommodation

Thus far, input panels have been described with the assumption that theuser input will be in a language that is written in rows from left toright and then those rows are written from top to bottom, such asEnglish. In that case, it is desirable that the input panel expandsrightward and then downward to match the way that English is written.However, not all languages are written in this manner. For example,Hebrew is written in rows from right to left and then those rows arewritten from top to bottom, and traditional Chinese is written from topto bottom in columns and then those columns are written from right toleft. Therefore, it may be desirable that the input panel expands indifferent directions than as previously described. For example, wherethe user is writing in Hebrew, the input panel may first expand leftwarduntil it reaches the Mhl margin, and then downward as needed for eachrow until it reaches the Mvb margin. The right and top edges of theinput panel would remain fixed, and the anchor point may be theintersection of the right and top edges. Also, in the case of Hebrewboundary 606 may be a left boundary. Or, where the user is writing inChinese, the input panel may first expand downward until it reaches theMvb margin, and then leftward as needed for each column until it reachesthe Mhl margin. The right and top edges of the input panel would againremain fixed. To generalize, the input panel may expand in two out ofthe four possible directions, and remain fixed in the other twodirections.

The particular language being used may depend upon the context in whichthe input panel is created, upon user commands, upon the operatingsystem, upon the input area associated with the input panel, and/or uponthe software application currently in focus. The buttons 1301 mayfurther allow the user to manually choose the language. The language mayfurther be chosen through interaction with the invocation target. Boththe manner in which the input panel expands and the manner in which theuser's handwriting is recognized may depend upon the language chosen.

CONCLUSION

Thus, a useful and convenient way to add user input in stylus-based andother computer systems has been described. An input panel is describedthat may dynamically expand in an on-demand manner that is particularlyeffective for receiving written or typed user input. The input panel mayprovide easy access to written or typed user input when and where neededby the end user, while reducing or minimizing the amount of valuablescreen “real estate” used and reducing or eliminating the possibility ofobscuring other elements displayed by the operating system and softwareapplications.

While apparatuses and methods as described herein embodying variousaspects of the present invention are shown by way of example, it will beunderstood that the invention is not limited to these particularembodiments. Modifications may be made by those skilled in the art,particularly in light of the foregoing teachings. For example, one ormore of the elements of the aforementioned embodiments may be utilizedalone or in combination with one or more elements of the otherembodiments. Also, although portions of the disclosure are described inconnection with Microsoft WINDOWS brand operating systems, it should beunderstood that other operating systems may be used, such as othergraphical user interface-based operating systems. In addition, theinvention has been defined using the appended claims, however theseclaims are also illustrative in that the invention conceived of includeselements and steps described herein in any combination or subcombination. It will also be appreciated and understood thatmodifications may be made without departing from the true spirit andscope of the invention.

1. In a computer, a method comprising steps of: displaying an inputpanel on a display extending along different first and second axes;receiving input into the input panel; and expanding the input panelalong the second axis responsive to the input extending along the firstaxis.
 2. The method of claim 1, wherein the first axis is a horizontalaxis, and the second axis is a vertical axis.
 3. The method of claim 1,wherein the first axis is a vertical axis, and the second axis is ahorizontal axis.
 4. The method of claim 1, wherein the input panelexpands along the second axis in only a single direction.
 5. The methodof claim 1, wherein the step of receiving the input includes receivinginput from a stylus upon a touch-sensitive display, and the step ofdisplaying includes displaying the input panel on the display.
 6. Themethod of claim 1, further including a step of expanding the input panelalong the first axis responsive to the input extending along the firstaxis.
 7. The method of claim 6, wherein the step of expanding the inputpanel along the first axis includes expanding the input panel along thefirst axis responsive to the input reaching a boundary within the inputpanel.
 8. The method of claim 1, wherein the step of expanding the inputpanel along the second axis responsive to the input extending along thefirst axis includes expanding the input panel downward responsive to theinput extending rightward.
 9. The method of claim 1, wherein the step ofexpanding the input panel along the second axis responsive to the inputextending along the first axis includes expanding the input panelleftward responsive to the input extending downward.
 10. In a computerhaving a stylus-sensitive input surface, a method comprising steps of:determining whether a stylus performs a first action relative to theinput surface; responsive to the stylus performing the first action,displaying an invocation target; and displaying an input panel afterdisplaying the invocation target.
 11. The method of claim 10, whereinthe first action is the stylus hovering over the input surface.
 12. Themethod of claim 10, wherein the input surface is also a display, and thesteps of displaying the invocation target and the input panel includedisplaying the invocation target and the input panel on the display. 13.The method of claim 10, wherein the step of displaying the invocationtarget includes displaying the invocation target at a location thatdepends upon the location of an insertion point.
 14. The method of claim10, wherein the step of displaying the invocation target includesdisplaying the invocation target at a location that depends upon thelocation of the stylus.
 15. In a computer having a stylus-sensitiveinput surface, a method comprising steps of: displaying a system prompt;determining whether a stylus performs a first action relative to theinput surface; determining whether the first action is associated withan input area that is of a first type or a second type; responsive tothe first action being associated with an input area of the first type,displaying an invocation target and displaying a first input panel afterdisplaying the invocation target; and responsive to the first actionbeing associated with an input area of the second type, displaying asecond input panel.
 16. The method of claim 15, wherein the first typeis an input area containing user input, and the second type is an inputarea devoid of user input.
 17. The method of claim 15, wherein the firstinput panel differs from the second input panel.